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Abstract 

An early-alert system (MavCLASS) was developed and piloted 
in a large gateway math class with 611 freshman students to identify 
academically at-risk students and provide alert messages. It was found 
that there was significant association between the alert messages stu¬ 
dents received and their visits to the university’s tutor center. Further, 
the achievement of students who visited the tutor center was im¬ 
proving over the semester. Evidence from the study suggests that an 
early-alert system focused on personalized feedback from instruction¬ 
al staff correlates with the help-seeking behaviors of at-risk students 
in large gateway classes. 
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Large classes of between 100 and 1000 have become common 
in higher education (Smith et al., 2005). Literature shows that large 
classes present many challenges to teaching and learning, including 
poor student engagement and low satisfaction (Gibbs, 1992). To ad¬ 
dress these challenges, much of the literature focuses on adapting the 
instruction mode from content-centered lectures to learner-centered 
activities. While effective classroom activities are critical, it seems 
self-evident that learning is optimized when students are also engaged 
in positive learning behaviors outside of the classroom, such as seek¬ 
ing help from the tutor centers. It is especially important for students 
from large courses to use the tutor centers, because the opportunities 
they get help directly from the instructor are so limited due to the 
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large class size. 

However, at our university, there was no formal mechanism to 
motivate students to seek help from the tutor center, Center for Ac¬ 
ademic Success (CAS). Additionally, there was no systematic process 
to track and assess the effects of CAS on student performance. 

To address the above challenges, we developed an early-alert 
system, called Maverick Comprehensive Learning Analytics Support 
System (MavCLASS), to encourage students to visit CAS. The Mav- 
CLASS project was just piloted in a large-cohort gateway class: Math 
098 Intermediate Algebra. The purpose of this study is to explore the 
patterns of student visits to CAS under the MavCLASS intervention 
and assess the relationship between the tutoring services provided by 
CAS and student performance in Math 098. 

Interventions to Increase Tutor Center Use 

Academic Tutor Centers are one method of improving student 
achievement and retention rates (Thompson, 2007). These centers 
often operate on an as-requested basis, where the onus is on the 
student to initiate contact. There are many factors associated with 
students’ willingness to seek help from the tutor center, including 
students’ motivation, self-esteem, and self-efficacy (Karabenick & 
Knapp, 1991), as well as many environmental factors (Lee, 2007). 

Bosco (2012) suggests that interventions designed to increase 
the frequency with which students seek help should begin early, 
follow up with students at several points in the semester, and dia¬ 
log about specific challenges and strategies relevant to the students. 
Bosco’s argument echoes with the perspective that personalized help 
could be effective ways to increase graduation and retention rates 
among college students (Capaldi, Lombardi, & Yellen, 2006). 

The emerging “big data” and analytics technologies in high¬ 
er education have provided new tools for developing personalized 
advising interventions. Student data can potentially inform university 
staff and faculty on students’ performance and provide students a 
mechanism by which they could involve themselves in developing 
more positive learning behaviors, such as seeking help from the tutor 
center (Hrabowski III, Suess, & Fritz, 2011). Dringus (2012) suggests 
that student data must be “measurable, visible and transparent” if it 
is to be valuable in informing academic interventions (p. 98). This 
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maps to broader theories of feedback and feedback interventions, 
which emphasize, among other things, that feedback must be seen 
by the recipient to be “legitimate, trustworthy, knowledgeable, and 
likeable” (Pintrich & Schunk, 2002, p. 168). 

Given this background, we felt that a successful early-alert 
system would have to include components that provide ongoing, per¬ 
sonalized feedback about student performance. Additionally, it would 
need to suggest clear actionable items so that students understand 
what to do to improve their performance. 

The Maverick Comprehensive Learning Analytics System 

(MavCLASS) 

The MavCLASS project was piloted in the course Math 098 
Intermediate Algebra in Fall 2013. The goal was to create a system 
to allow faculty and graduate assistants (GAs) to view students’ 
performance in greater detail and develop personalized feedback to 
encourage learners to seek help from CAS for course improvement. 
The project had three components: standard-based formative assess¬ 
ments, data dashboards, and personalized alert messages. 

When designing the course, the instructor worked with an 
instructional designer to create weekly standards students were ex¬ 
pected to achieve, and then organize the course content and assess¬ 
ments around these standards. With this approach, each assessment 
(e.g., homework, quiz, test) was associated with a few specific course 
standards so that faculty and GAs could quickly identify the specific 
standards students needed to work on. 

The data dashboard worked across two assessment systems, 
including the university’s Learning Management System (LMS) that 
provided exam and class participation scores and a publisher system 
that managed assignments and quizzes (Cengage’s WebAssign). These 
data were cleaned, analyzed and displayed in colors of green, yellow, 
or red for the instructor and GAs to review. The colors were deter¬ 
mined based on algorithms defined by the instructor to reflect stu¬ 
dent assessment achievement levels. Students who got the yellow and 
red colors were identified as in the cautionary and danger of failing 
and would receive alert messages from the GAs of the course. 

The alert messages were sent out to students within 1 week 
after the assessment scores were published on their dashboards. 
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The message began from a standardized script: It told students their 
current status on the assessments and encouraged them to seek help 
from CAS. The GAs were then instructed to manually customize 
the alert messages and send them out to students. Since each assess¬ 
ment was associated with a few specific standards, by looking at the 
dashboard, the GAs could quickly identify the course standards the 
student was struggling with, and therefore, they could explicitly point 
out to students the associated learning materials they should work on, 
including the lecture notes, textbook chapters and exercises. In the 
alert message, the students were instructed to bring these suggested 
learning materials to CAS so that there was a clear focus during the 
tutoring sessions. 

Research questions 

This study aims to answer the following questions: 

1. Under the MavCLASS intervention, is there any pattern of 
student visits to CAS? 

2. Is there any relationship between the alerts and students’ 
visits to CAS? 

3. Is there any relationship between the student visits to CAS 
and their achievement? 

Methods 

Data Collection 

Three types of data were collected from the 611 students 
who took Math 098 in Fall 2013: the alert message data, the student 
achievement data, and the CAS visit data. The alert message data 
were collected through MavCLASS. In Math 098, students who per¬ 
formed below the standards on any assignment, quiz or exam would 
be considered as at-risk students. Their scores on these assessments 
would be displayed in yellow or red and the alert messages were sent 
out to these students. We reviewed the scores for all types of assess¬ 
ments in MavCLASS to identify the recipients of the alert messages, 
as well as the dates when the alerts were sent out. 

The student achievement was mainly measured by the four 
high-stake exams in this course. Students took these exams at Week 5, 
Week 9, Week 13, and Week 16 of the semester. Student performance 
data on the four exams were retrieved from MavCLASS. 

The CAS visit data were collected at the end of the semester. 
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The data identified the students who visited CAS for Math 098 in Fall 
2013 and the time and date of their visits. 

Data Analysis 

The data analysis process included the following steps. First, 
relationships were established among the three sets of raw data based 
on the student identification information. Data was subsequendy 
anonymized and rescaled by converting the student assessment scores 
to the accuracy rate (i.e., the total of points earned divided by the 
total of points possible). Microsoft Excel Version 14.4.1 and IBM 
SPSS Statistics 20 were used to conduct various descriptive analyses 
and t test. 

Results 

Research Question 1: Patterns of CAS Visits 

Figure 1 shows that 478 students (78.2%) from Math 098 
received alert messages. This means that these students did not meet 
the standards on at least one assessment of the course and were en¬ 
couraged to visit CAS to seek help. Among these students, 81 visited 
CAS, but 397 did not. 133 students (21.8%) received no alert mes¬ 
sages throughout the semester, suggesting that they performed above 
the standards on every assessment. Twelve of these higher-perform¬ 
ing students still visited CAS, even if they were never prompted to do 
so by MavCLASS. 


= Received alert; Visited 
111 Received Alert; Not visited 
* Received no alert; Visited 
^Received no alert; Not visited 



Figure 1. MavCLASS students divided into four groups. 

Figure 2 shows the distribution of the students’ CAS visits in 
Fall 2013. From Week 6 to Week 9, when the students took the first 
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and second high-stake exams, 581 alert messages were sent out and 
there were 145 visits to CAS, representing 45% of the total visits 
throughout the semester. Between Week 10 and Week 13, 598 alert 
messages were sent. In contrast, only 38 CAS visits occurred during 
that time, representing 11.7% of the total visits. 

This pattern indicates that the students were much more 
engaged in help-seeking in the second quarter of the semester (i.e., 
between Week 6 and Week 9). Despite the increase in the number 
of alert messages in next four weeks, the students did not visit CAS 
as much as they did prior to Week 9. This finding is consistent with 
previous research (Bevitt, Baldwin, & Calvert, 2010), which has 
confirmed that early assessments and interventions (e.g., alert mes¬ 
sages) are effective methods of engaging students in positive learning 
behaviors, such as seeking help from the tutor centers. 


Number of CAS 



Week 


Research Question 2: Relationship between MavCLASS and 
CAS Visits 

A t-test was conducted between students who received alert 
messages and students who did not receive any alert to determine 
whether there is any difference in their visits to CAS. Table 1 in¬ 
dicates that the difference between the two groups is significant 
(p<.01). Because the students received the alerts when they failed to 
meet the standards on at least one assessment, the t test results could 
be interpreted in at least two ways. First, the students with lower 
assessment outcomes tend to visit CAS more often. Second, the 
students receiving the alerts are more likely to visit CAS. Admittedly, 
based on the current data, it is premature to determine any causal 
relationship between the alerts and the students’ visits to CAS. But 
these findings are consistent with the notion that even the simple 
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notification interventions (e.g., letting students know their assessment 
grades) may lead to positive changes in student learning behaviors 
(Jayaprakash, Moddy, Lauria, Regan, & Baron, 2014). 

Table 1 


t Test between Students Who Received Alerts and Students Who Did Not 




F 

Sig. 

t 

df 

Sig. 

(2-tailed) 

Number 

of CAS 

Visits 

Equal 

Variances 

assumed 

15.106 

.000 

2.169 

609 

.031* 


Equal 

Variances 

not 

assumed 



3.101 

436.991 

.002** 


Note. *p<.05, two-tailed. **p<.01, two-tailed 


Research Question 3: Relationship between CAS Visits and 
Achievement 

In Math 098, the students took four high-stake exams through¬ 
out the semester. Student achievement on these exams was compared 
between two groups. Group 1 consists of 518 students who did not 
visit CAS. Group 2 includes 93 students who visited the tutor center. 

Figure 3 shows that, the Group 1 students achieved nearly 70% 
of accuracy rate on Exam 1, but their performance was continu¬ 
ously declining on the subsequent exams. For the Group 2 students, 
their average accuracy rate was about 63% on Exam 1, but slighdy 
increased to nearly 65% on Exam 2. After that, their accuracy rate 
decreased to 64% on Exam 3 and to 57% on the final exam. 

Generally, Group 1 had better performance than Group 2 
on these exams. However, the achievement gap, as reflected by the 
difference in the average accuracy rate between the two groups, was 
nearly 8% on Exam 1, but was getting closer and closer. Eventually, 
the two groups had about the same performance on the final exam. 
The t test shows that the difference between the two groups was 
significant (p <.005) on Exam 1 but no significant between-group 
difference was detected on the other exams. The diminishing trend 
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of the achievement gap is more clearly reflected in Figure 4, which 
presents the students’ z scores. On the first exam, the gap between 
the two groups was about .33 standard deviation (SD). But the gap 
was getting much closer on the following exams and got to less than 
.03SD on the final exam. 



■Group I 
■Group 2 


Figure 3. Average accuracy rates of Group 1 and Group 2 students. 



■Group 1 
■Group 2 


Figure 4. z scores of Group 1 and Group 2 students. 


The two groups of students were further divided into four sub¬ 
groups based on whether they received any alert messages. As men¬ 
tioned earlier, students who never received any alert passed all the 
course assessments, and therefore, were considered higher-perform¬ 
ing students. Students who received the alert messages failed at least 
one assessment, and were identified as lower-performing students. As 
the Figure 5 suggests, among the higher-performing groups, students 
who visited CAS generally had higher achievement than those who 
did not and their final exam scores were increased from the earlier 
exams. In contrast, the performance of higher-performing students 
who did not visit CAS was decreasing across the four exams. 

























Early-Alert System | 69 


A similar pattern was found for the lower-performing groups. 
Students who visited CAS were more than .4SD below the class’s 
average level on the first exam, but they were making improvements 
and increased their achievement by over .2SD at the end of the se¬ 
mester. However, the other subgroup, the lower-performing students 
who did not visit the center, did not make any progress. Their per¬ 
formance was around .2SD below the average level for each of the 
exams over the semester. 



Figure 5. z scores of four subgroups of students. 


Discussion 

Results from this study, along with the previous studies (Bevitt, 
Baldwin, & Calvert, 2010; Colby, 2004), indicate that it is important 
to provide students with meaningful assessments and feedback early 
in the semester in order to encourage their use of tutor center for 
academic improvement. As Pistilli and Arnold (2010) point out, often 
students do not understand how well they are performing in a class 
until it is too late to make any positive changes. 

This study also suggests that early interventions have the po¬ 
tential to positively impact student academic performance through 
increasing help-seeking behaviors. In this study, the early-alert sys¬ 
tem seems positively associated with the student visits to CAS. This 
finding is consistent with the work at Purdue University, which shows 
that the use of relatively simple notification can have a significant im¬ 
pact on student behaviors which can lead to improved achievement 
(Jayaprakash, Moddy, Lauria, Regan, & Baron, 2014). 

Another contribution of the study is that it has built connec¬ 
tions between the students and the tutor center at the university. 
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Many institutions provide services tailored to their student needs, 
such as various tutoring sessions. Unfortunately, these services are 
often underused by students who could benefit from them the most 
(Tinto, 2012). The intervention piloted in this study has the potential 
to address this issue by identifying the academically at-risk students 
and sending alert messages on an ongoing basis to connect the tutor 
center with those students. 

Additionally, the personalized alert messages received by stu¬ 
dents could potentially drive the content of these tutoring sessions, 
making them more productive and manageable, particularly for those 
students in large gateway classes. The rise in achievement for those 
students who sought help from CAS would seem to indicate that, in 
general, students who seek help from CAS are able to improve their 
academic achievement. 

Limitations 

As with any study, limitations existed with this study. Due to 
the unavailability of the CAS visit data from the previous years, the 
causal effects of MavCLASS were unable to be determined. Since 
CAS does not have any records before Fall 2013, it is not yet possible 
to track the students’ CAS visits over time to see whether there is any 
difference before and after the MavCLASS implementation. 

Another limitation of the study is that only 15.2% of students 
sought help from CAS for their math course, but the current data 
cannot help us understand why the majority of students did not visit 
CAS. Researchers find that, students’ psychological constructs, such 
as self-efficacy (Ryan & Pintrich, 1997) and goal orientations (New¬ 
man, 1998) may play a role in their help-seeking behaviors. These 
factors will need to be considered in the next phase of the project 
so as to design effective interventions. For example, we could send 
customized feedback messages that match students’ goal orientations 
to better motivate the students to use the tutor center. 

Conclusion 

In this study, we piloted the MavCLASS intervention in a large 
gateway course. MavCLASS functioned as a systematic mechanism 
that established direct connections between the students and the 
tutor center (i.e., CAS) at the university. Additionally, the study gen¬ 
erated empirical evidence regarding the pattern of the student visits 
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to CAS across the semester and the relationship between the student 
achievement and their CAS visits. As discussed above, the findings 
echoed with previous studies and provided implications for the de¬ 
sign and implementation of feedback interventions that increase the 
visits to the tutor center. 
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